Qwen2.5-VL Technical Report
Table of Contents
1. Model Architecture
1.1. MLP-Based Vision-Language Merger
To address efficiency problems caused by long sequence of image features, Qwen2.5VL first group adjacent \( 2\times 2 \) patch features, concatenate them, then pass them through a two-layer MLP to align dimension with text token embeddings. When training LLM backbone, vision embeddings are appended to text token embeddings.
1.2. Vision Encoder
Qwen2.5 of different sizes (3B, 7B, 72B) uses the same vision encoder, which is a modified version of ViT.
1.2.1. Patch Embedding for Images and Videos
To handle videos, Qwen2.5VL uses \( 14\times 14 \) as spacial unit, and extends to \( 2\times 14\times 14 \), which means for videos, 2 consecutive frames are used to form a single temporal token.
1.2.2. Window Attention
In Qwen2.5VL, concerning computational efficiency, only 4 layers are full attention. The rest are window attention without padding such that each window covers adjacent \( 112 \times 112 \) pixels, equivalent to \( 8\times 8 \) surrounding patches.
The design of 4 layers of efficiency is for exchanging patch information globally to trade off information loss due to computational efficiency.
This design is similar to Convolutional NN, so it leverages the advantage that it can handle varying input resolution.
1.2.3. Multimodal Positional Embedding
To represent video frames, we need 3D information: \( (t, h, w) \) representing temporal timestamp, height, width respectively.
This 3D representation does not break 1D RoPE and 2D RoPE. For text, \(p\)-th token can be expressed as \( (p,p,p) \); for images, the time dimension remains static; for videos, time dimension uses absolute timestamp relative to the video for better temporal understanding.
1.2.4. Other Components
To align with LLM backbone, Qwen2.5VL uses RMSNorm for normalization and SwiGLU as activation.